Ultrasound Analytics for Actionable Information

ABSTRACT

Systems and techniques are described for gathering information on the health of individuals trapped in an accident to provide actionable information to a first responder system. In some implementations, a monitoring system monitors a property that includes sensors located at the property and generate first sensor data. A monitor control unit receives the first sensor data and generates an alarm event for the property based on the first sensor data. Based on generating the alarm event for the property, the monitor control unit dispatches an autonomous drone. The autonomous drone is configured to navigate the property. Using an onboard sensor, the autonomous drone generates second sensor data. Based on the second sensor data, the autonomous drone determines a location within the property where a person is likely located. The autonomous drone provides, for output, data indicating the location within the property where the person is likely located.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/591,920 filed Nov. 29, 2017, and titled “Ultrasound Analytics forActionable Information,” which is incorporated herein by reference.

TECHNICAL FIELD

This specification relates generally to integrated security technology,and in particular, to integrated security technology to provideactionable information to first responders using ultrasound data.

BACKGROUND

Integrated security includes the use of security hardware in place on aproperty, such as a residential property and a commercial property.Typical uses of security at a particular property includes detectingintrusion, detecting unlocked doors, detecting when an individual isharmed at the property, and tripping one or more alarms.

SUMMARY

The subject matter of the present disclosure is related to systems andtechniques for gathering information on the health of one or moreindividuals trapped in an accident to provide actionable information toa first responder system. The techniques may use ultrasound, cameraimages, GPS locational data, and machine learning algorithms to providethe actionable information to the first responder system. The machinelearning algorithms may include algorithms such as one or more neuralnetwork models, Bayesian learning models, or any other type of machinelearning technique, to detect injuries of the individuals trapped in theaccident. In response to detecting injuries of the individuals trappedin the accident, the systems may transmit a notification to a firstresponder system and other individuals that may know the injuredindividual indicating the individual is trapped and injured. The benefitof providing the indication of the injured individual is such that otherindividuals related to the injured individual can be aware of the statusof the injured individual in the case of an emergency, such as a fire,earthquake, or flood, to name a few examples. Additionally, by notifyingthe first responder system, the first responder system can take one ormore steps to save the injured individuals when time is of the essenceand the injured individual's life is in severe condition. The one ormore steps may include pinpointing the location of the injuredindividual at a facility that has toppled due to a natural disaster whenfinding the injured individual is next to impossible with the human eyealone, notifying one or more other individuals of the injuredindividual's status and location, and determining the injury of theinjured individual in an efficient manner to provide the correct care.

In some implementations, the techniques may utilize a set of sensorsincluding a camera (or an array of cameras), a Global Positioning System(GPS) device, and an ultrasound transducer. Each of the sensors may beco-located and mounted on one unit, such as a plane or drone. Thesensors can communicate to a backend over a WiFi or cellularcommunication network. In some implementations, the backend isresponsible for transforming the data provided by each of the cameras,the GPS device, and the ultrasound transducer into one or more varioustypes of data and performing advanced analytics on the various types ofdata. In some implementations, the backend is responsible foraggregating the various types of data into an aggregated map thatincorporates all usable information provided by the camera, the GPSdevice, and the ultrasound transducer. The backend may provide theaggregated map to a first responder system such that the first respondersystem can identify and prioritize providing actionable rescue teams forthe identified individuals.

In one general aspect, a method is performed by one or more computers ofa monitoring system. The method includes generating, by one or moresensors of a monitoring system that is configured to monitor a property,first sensor data; based on the first sensor data, generating, by themonitoring system, an alarm event for the property; based on generatingthe alarm event for the property, dispatching, by the monitoring system,an autonomous drone; navigating, by the autonomous drone of themonitoring system, the property; generating, by the autonomous drone ofthe monitoring system, second sensor data; based on the second sensordata, determining, by the monitoring system, a location within theproperty where a person is likely located; and provide, for output bythe monitoring system, data indicating the location within the propertywhere the person is likely located.

Other embodiments of this and other aspects of the disclosure includecorresponding systems, apparatus, and computer programs, configured toperform the actions of the methods, encoded on computer storage devices.A system of one or more computers can be so configured by virtue ofsoftware, firmware, hardware, or a combination of them installed on thesystem that in operation cause the system to perform the actions. One ormore computer programs can be so configured by virtue havinginstructions that, when executed by data processing apparatus, cause theapparatus to perform the actions.

Implementations may include one or more of the following features. Forexample, in some implementations, based on navigating the property,generating, by the monitoring system, a map of the property; andproviding, by the monitoring system, for output, data indicating thelocation within the property where the person is likely located byproviding, for output, the map of the property with the location wherethe person is likely located.

In some implementations, the method further includes determining, by themonitoring system, that the person is likely injured based on secondsensor data; and providing, for output by the monitoring system, dataindicating that the person is likely injured.

In some implementations, the method further includes based ondetermining that the person is likely injured, generating, by theautonomous drone of the monitoring system, using an additional onboardsensor, third sensor data; based on the third sensor data, determining,by the monitoring system, a severity of the injury to the person; andproviding, for output by the monitoring system, the data indicating thatthe person is likely injured by providing, for output, the dataindicating that the person is likely injured and data indicating theseverity of the injury to the person.

In some implementations, the method further includes the onboard sensoris a camera and the second sensor data is image data, and the additionalonboard sensor is an ultrasound sensor and the third sensor data isultrasound data.

In some implementations, the method further includes providing, by themonitoring system, second sensor data as an input to a model trained toidentify locations of people; and determining, by the monitoring system,the location within the property where a person is likely located basedon an output of the model trained to identify locations of people basedon the second sensor data.

In some implementations, the method further includes receiving, by themonitoring system, labeled training data that includes first labeledsensor data that corresponds locations with people and second labeledsensor data that corresponds to locations without people; and training,by the monitoring system, using machine learning, the first labeledsensor data, and the second labeled sensor data, the model to identifylocations of people based on the second sensor data.

In some implementations, the method further includes based on the secondsensor data, determining, by the monitoring system, that the person islikely alive; and providing, by the monitoring system, for output, dataindicating that the person is likely alive.

In some implementations, the method further includes the second sensoris a microphone and the second sensor data is audio data, and the methodincludes providing, by the monitoring system, the audio data as an inputto a model trained to identify human sounds; and determining, by themonitoring system, that the person is likely alive based on an output ofthe model trained to identify human sounds.

In some implementations, the method further includes based ondetermining a location within the property where a person is likelylocated, activating, by the monitoring system, a communication channelbetween a device outside the property and the autonomous drone.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a contextual diagram of an example system of an integratedsecurity environment for detecting one or more injured individuals at amonitored facility.

FIG. 2 is a contextual diagram of an example system of a buildingdestruction environment for detecting one or more injured individuals.

FIG. 3 is a contextual diagram of an example system for training aneural network model for ultrasound analytics.

FIG. 4 is a flowchart of an example process for providing datacorresponding to a detected individual for ultrasound analytics.

FIG. 5 is a flowchart of an example system for processing datacorresponding to a detected individual for ultrasound analytics.

FIG. 6 is a block diagram of an example integrated security environmentfor ultrasound analytics that may utilize various security components.

DETAILED DESCRIPTION

FIG. 1 is a contextual diagram of an example system 100 of an integratedsecurity environment for detecting one or more injured individuals at amonitored facility. Though system 100 is shown and described including aparticular set of components in a monitored property 102 includes acontrol unit server 104, network 106, cameras 108, lights 110, sensors112, home devices 114, security panel 126, drone 130, network 134,remote processing unit 136, and first responder system 140, the presentdisclosure need not be so limited. For instance, in someimplementations, only a subset of the aforementioned components may beused by the integrated security environment for monitoring the controlunit servers in each monitored property. As an example, there may be asystem 100 that does not use the lights 110. Similarly, there may beimplementations that the control unit, such as control unit server 104,is stored in the remote processing unit 136. Yet other alternativesystems also fall within the scope of the present disclosure such as asystem 100 that does not use a control unit server 104. Rather, thesesystems would communicate directly with the remote processing unit 136to perform the monitoring. For these reasons, the system 100 should notbe viewed as limiting the present disclosure to any particular set ofnecessary components.

As shown in FIG. 1, a residential facility 102 (e.g., home) of user 118is monitored by a control unit server 104 that includes componentswithin the residential facility 102. The components within theresidential facility 102 may include one or more cameras 108, one ormore lights 110, one or more sensors 112, one or more home devices 114,and the security panel 126. The one or more cameras 110 may includevideo cameras that are located at the exterior of the residentialfacility 102 near the front door 116, as well as located at the interiorof the residential facility 102 near the front door 116. The one or moresensors 112 may include a motion sensor located at the exterior of theresidential facility 102, a front door sensor that is a contact sensorpositioned at the front door 116, and a lock sensor that is positionedat the front door 116 and each window. The contact sensor may sensewhether the front door 118, the garage door, or the window is in an openposition or a closed position. The lock sensor may sense whether thefront door 116 and each window is in an unlocked position or a lockedposition. The one or more home devices 114 may include home appliancessuch as a washing machine, a dryer, a dishwasher, an oven, a stove, amicrowave, and a laptop, to name a few examples. The security panel 126may receive one or more messages from a corresponding control unitserver 104 and a remote processing unit 136.

The control unit server 104 communicates over a short-range wired orwireless connection over network 106 with connected devices such as eachof the one or more cameras 108, one or more lights 110, one or more homedevices 114 (washing machine, a dryer, a dishwasher, an oven, a stove, amicrowave, a laptop, etc.), one or more sensors 112, the drone 130, andthe security panel 126 to receive sensor data descriptive of eventsdetected by the one or more cameras 108, the one or more lights 110, thedone 130, and the one or more home devices 114 in the residentialfacility 102. In some implementations, each of the connected devices mayconnect via Wi-Fi, Bluetooth, or any other protocol used to communicateover network 106 to the control unit server 104. Additionally, thecontrol unit server 104 communicates over a long-range wired or wirelessconnection with a remote processing unit 136 over network 134 via one ormore communication links. In some implementations, the remote processingunit 136 is located remote from the residential facility 102, andmanages the monitoring at the residential facility 102, as well as other(and, perhaps, many more) monitoring systems located at differentproperties that are owned by different users. In other implementations,the remote processing unit 136 communicates bi-directionally with thecontrol unit server 104. Specifically, the remote processing unit 136receives sensor data descriptive of events detected by the sensorsincluded in the monitoring system of the residential facility 102.Additionally, the remote processing unit 136 transmits instructions tothe control unit server 104 for particular events.

In some implementations, a user 118 may install a device to monitor theremote property 102 from the outside. For instance, the user 118 mayinstall a drone 130 and a corresponding charging station 142 to monitorthe activity occurring outside and inside the residential property 102.In some implementations, the control unit server 104 may detect when thedrone 130 has departed from the charging station 142. The drone 130 mayautomatically depart from the charging station 142 at predeterminedtimes set by the user 118 according to a signature profile. Oncedeparted from the charging station 142, the drone 130 may fly apredetermined path 132 as set by the user according to a profile. Thepredetermined path 132 may be any path around the residential property102 as described by the signature profile. The signature profile will befurther explained below.

In some implementations, the drone 130 will have a set of devices 131for providing sensor data to the control unit 104. The set of devices131 may include a camera or an array of cameras, a GPS device, and anultrasound transducer, to name a few examples. The drone 130 mayinstruct the set of devices 131 to record and monitor while the drone130 flies the predetermined path 132.

In the example shown in FIG. 1, user 118 may be in the residentialfacility 102 and can arm the residential facility 102 at any point intime. In doing so, the user 118 may turn off each of the one or morelights 110, turn off each of the one or more home devices 114, lock thefront door 116, and close and lock each of the one or more windows. Theuser 118 may interact with a client device 120 to activate a signatureprofile, such as “arming home” for the residential facility 102.Alternatively, the user 118 may keep the one or more lights 110 on, keepthe one or more home devices 114 on while setting the “arming home”profile.

In some implementations, the client device 120 may display a webinterface, an application, or a device specific application for a smarthome system. The client device 120 can be, for example, a desktopcomputer, a laptop computer, a tablet computer, a wearable computer, acellular phone, a smart phone, a music player, an e-book reader, anavigation system, a security panel, or any other appropriate computingdevice. In some implementations, the client device 120 may communicatewith the control unit server 104 over the network 106. The network 106may be wired or wireless or a combination of both and can include theInternet.

In some implementations, user 118 may communicate with the client device120 to activate a signature profile for the residential facility 102. Toillustrate, user 118 may first instruct the control unit server 104 toset a signature profile associated with arming the residential facility102. For example, user 118 may use a voice command to say “Smart Home,arm house,” to the client device 120. The voice command may include aphrase, such as “Smart Home” to trigger the client device 120 toactively listen to a command following the phrase. Additionally, thephrase “Smart Home” may be a predefined user configured term tocommunicate with the client device 120. The client device 120 can sendthe voice command to the control unit server 104 over the network 106.

In some implementations, the control unit server 104 may notify theremote processing unit 136 that the residential facility 102 is to bearmed. In addition, the control unit 104 may set associated parametersin response to receiving the voice command. Moreover, the control unit104 can send back a confirmation to the client device 120 in response toarming the residential facility 102 and setting the associatedparameters. For example, the control unit server 104 may transmit aresponse to the client device 120 that reads “Smart Home armed.”

In some implementations, in order for the control unit server 104 toallow user 118 and others to set and activate a signature profile casefor the residential facility 102, the user 118 and others may define andstore signature profiles in the control unit server 104. In otherimplementations, the user 118 and others may define and store signatureprofiles in the remote processing unit 136. The signature profile may beassociated with each user and allow for various use cases of the devicesin the residential facility 102. Each of the signature profiles can beassociated with one user, such as user 118 or user 124. For example, auser 118 may create a signature profile for arming the residentialfacility 102. In another example, a user 122 may create a signatureprofile for monitoring the residential facility 102 with a drone 130 formonitoring the residential facility 102.

In some implementations, user 122 may store one or more parametersassociated with a use case in his or her signature profile.Specifically, the one or more parameters for each use case may describea volume level in decibels (dB) of the speakers 108, an aperture amountfor the cameras 110, a brightness intensity level of the lights 112,turning on home devices 117 such as television, laptop, one or morefans, setting a specific temperature of a thermometer, opening orclosing the shades of a window a particular amount, alarm settingscorresponding to the security panel 126, defining a predetermined pathand a length of time for the drone 130 to monitor the residentialfacility 102, and any other parameters to describe the use case. Forexample, user 122 may create a signature profile with a use case for“arm home”. The user 122 may define a volume level of 0 dB for thespeakers 108, an aperture of f/16 for the one or more cameras 110, zerolumens for the one or more lights 112, turning off a television, turningoff a laptop, turning on fans, setting the thermometer to 67 degreesFahrenheit, fully closing the blinds of the one or more windows, andsetting the security panel 126 to notify the remote processing unit 136for any detected alarms.

In some implementations, the user 118 may define a predetermined path132 for the drone 130 to monitor around the residential facility 102.The predetermined path 132 may be drawn by the user 118 throughinteraction with the smart home application on the client device 120.The user 118 may additionally define the height and speed in which thedrone 130 flies around the residential property 102. For instance, theuser 118 may draw a circle on a map provided by the smart homeapplication on the client device 120, set the altitude to 10 feet, andset the drone 130's flying speed to 15 miles per hour. The user 118 candefine a period of time for the drone to monitor the residentialproperty 102. For example, the user 118 may enter the time of 1 hourinto the smart home application on the client device 120. Following thetime period in which the drone 130 monitors the residential property102, the user 118 can instruct the drone to return to the chargingstation 142 or to traverse a new predetermined path around residentialproperty 102, different from predetermined path 132.

In some implementations, the control unit server 104 sets the parametersfor the signature profile when the user 122 speaks “Smart home, armingthe home” to client device 120. The control unit server 104 saves theparameters in memory defined by the user 118 in the smart homeapplication on the client device 120 in response to the user setting theparameters. In addition, the control unit server 104 may transmit theset parameters for the signature profile to the remote processing unit136 to save for backup purposes.

In some implementations, the control unit server 104 may increase thesensitivity corresponding to each of the one or more sensors 114 for the“arming the home” use case. Specifically, control unit server 104 mayincrease the sensitivity for the front door sensor, the garage doorsensor, and the lock sensor by a predetermined factor so that smallermovements of the front door or garage door trigger an alarm event. Forexample, the sensitivity may be increased by a factor of five.

In some implementations, the control unit server 104 may send a responseto display a message on the client device 120 that says “Smart Home,home armed” once the control unit server 104 sets the parameters. Thecontrol unit server 104 may also transmit the same response to thedisplay 128 of security panel 126 once the control unit server 104 setsthe parameters. In addition, the control unit server 104 may transmit amessage to the remote processing unit 126 that the residential facility102 finished arming.

In some implementations, the drone 130's set of devices 131 may seek todetect the health of one or more individuals inside the residentialfacility 101. In particular, the set of devices 131 may gatherinformation on the health of the one or more individuals inside theresidential facility 102. As the drone 130 flies around the residentialfacility 102, the drone 130 scans areas external and internal to theresidential facility 102. In particular, the drone 130 may scan areas inproximity to the residential facility 102, scan through the walls of theresidential facility 102 to see the interior of the residential facility102, and monitor each level of the residential facility 102. The drone130 uses local machine learning algorithms along with ultrasound data,images, and GPS locational data captured by the set of devices 131 todetect one or more individuals in the residential facility 102. Shouldthe drone 130 detect an individual in the residential facility 102, thedrone 130 may move closer to the individual to perform a more detailedscan. The drone 130 then sends the captured data to the control unitserver 104 for further processing to determine the health of the one ormore individuals. The control unit server 104 may also acquire sensordata from the cameras 108, the lights 110, the sensors 112, and the homedevices 114 in response to receiving the captured data from the drone130. The control unit server 104 provides the captured data and thesensor data to the remote processing unit 136 for further processing adetermination of whether a first responder system 140 should becontacted.

For example, during stage (A), the user 118 sets the parameters for the“arming home” signature profile that includes a time for the drone toinitiate monitoring the residential property 102. At the set time asdesignated by the “arming home” signature profile, the control unitserver 104 sends an indication to the drone 130 via network 106 toinitiate monitoring the residential facility 102. The indication mayinclude GPS coordinates of the predetermined path, the length of time totravel, and the altitude or varying altitude around the residentialfacility 102 in which to travel. In some implementations, the remoteprocessing unit 136 may send an indication to the control unit server104 to instruct the drone 130 to initiate the monitoring of theresidential facility 102. In response to receiving the indication, thedrone 130 powers on, flies away from the charging station 142, and fliesthe predetermined path 132 as set in the “arming home” signatureprofile. During flight, the drone 130 uses the set of sensors 131 todetect one or more individuals in the residential facility 102.

In some implementations, the control unit server 104 may use the cameras108, the lights 110, the sensors 112, and the home devices 114 inconjunction with the set of sensors 131 to detect one or moreindividuals in the residential facility 102. For instance, as the drone130 travels around the predetermined path 132, the drone 130 may sendGPS coordinate updates to the control unit server 104. The control unitserver 104 may turn on one or more of the lights 110 in one or moreareas currently being viewed by the drone 130 to improve detectability.In addition, the control unit server 104 may increase sensitivity of oneor more sensors 112 in the one or more areas currently being viewed bythe drone 130 to also improve detectability. Should a motion detectorfrom the one or more sensors 112 detect movement in an area of theresidential facility 102, the control unit server 104 can transmit a GPScoordinate of the detected motion sensor to the drone 130 to focus theset of devices 131 on the area designated by the transmitted GPScoordinate. The GPS coordinate may be inside or outside the residentialfacility 102.

During stage (B), the drone 130 detects an individual in the residentialfacility 102. For instance, the set of devices 131 captures data duringthe drone 130's flight around the predetermined path 132. The dataincludes camera images and GPS locational data. The drone 130 feeds thecamera images and the GPS locational data to a local processing engineincluded in the drone 130's memory. The local processing engine producesan indication that an individual has been detected in the camera images.In response to determining that an individual, such as user 118, hasbeen detected, the drone 130 moves closer to that individual to performan ultrasound scan. The drone 130 may move closer to a window of theresidential facility 102 or closer to a wall of the residential facility102 to perform the ultrasound scan. The drone 130 may perform anultrasound scan of the user 118 at different portions of the user 118'sbody. For instance, the drone 130 may initiate scanning user 118's head,then move to scan the user 118's shoulder, and down to user 118's feet.These ultrasound scans will be used later in constructing a mappedenvironment of the user 118.

During stage (C), the drone 130 detects another individual, such as user124, in the residential facility 102. The drone 130 performs similarsteps as described in stage (B) to detect user 124. In someimplementations, the local processing engine in the drone 130 producesan indication of a detected person. In other implementations, the localprocessing engine in the drone 130 may produce a recognition of adetected person. For instance, based on the training of the localprocessing engine, the local processing engine may produce an indicationthat a person has been detected or that the person detected is user 124or Bob. This indication will be further described below.

During stage (D), the drone 130 provides the captured drone data 133 tothe control unit server 104 over the network 106. The captured dronedata 133 includes the captured images, the GPS locational data, and theindication provided by the local processing engine. The control unitserver 104 receives the captured drone data 133. The control unit server104 combines the captured drone data 133 with data provided by the oneor more cameras 108, the one or more lights 110, and the one or moresensors 112. For instance, the control unit server 104 may packagetogether the captured drone data 133 with images and video from thecameras 108, a brightness level from the one or more lights 110, andmotion or contact data from the one or more sensors 112 when a detectionwas made by the drone 130. In addition, the control unit server 104 mayinclude the data changes indicating the brightness level of the one ormore lights 110 and the sensitivity changes of the one or more sensors112 to improve detectability for the drone 130. This change data mayfacilitate the remote processing unit 136 in determining typical pathsof one or more individuals in the residential facility 102. This can beused to update the predetermined path 132 of the drone 130 for improvedtracking of individuals. Once the control unit server 104 packages thedata, the control unit server 104 transmits the packaged data as sensordata 135 to the remote processing unit 136.

During stage (E), the remote processing unit 136 receives the sensordata 135. The remote processing unit 136 includes a remote processingengine to produce an indication of the health of the individual detectedin the captured image. For instance, the remote processing engine of theremote processing unit 136 includes one or more machine learningalgorithms that can produce an indication of an injury of the individualfrom the sensor data 135. The injuries may include one or more brokenbones, external bleeding, and burn marks, to name a few examples. Theindication output by the remote processing unit 136 may include an imagefrom the ultrasound data including the detected individual and a taggeddescription of the injury. The remote processing engine provides theimage and the tagged description of the injury to a severity indicator.The severity indicator tags the input with a number indicating theseverity of the individual's health in the attached image. For example,as illustrated in FIG. 1, the control unit server 104 may provide sensordata 135 of two detected individuals in residential facility 102, user118 and user 124. The remote processing engine of the remote processingunit 136 may produce a severity indication of zero, corresponding to oneor more images from the ultrasound data of user 118. The severityindication of zero indicates that user 118 has no injury or appears tohave no injury. Likewise, the remote processing engine may produce aseverity indication of ten, corresponding to one or more images from theultrasound data of user 124, indicating a severe injury. The remoteprocessing engine may detect that user 124 has broken his arm, asillustrated by the images in the ultrasound data.

During stage (F), the remote processing engine provides a notificationto the owner of the residential facility 102. The notification includesone or more images and the corresponding severity of an injury of anidentified individual in each of the one or more images. In someimplementations, the remote processing engine in the remote processingunit 136 provides the notification to the client device 120 of user 118.The client device 120 may display the one or more images and thecorresponding severity of the injury of the identified individual ineach of the one or more images to the user 118. For example, theseverity of the injury may include a number such as ten or display amessage that recites “User Broke Arm” 122, as illustrated in FIG. 1. Theuser 118 may proceed to locate the injured individual, user 124, toprovide emergency assistance.

During stage (G), the remote processing engine provides a notificationto a first responder system 140. The notification includes areconstructed mapped environment of the images of the ultrasound scansand a corresponding severity indicator for each of the images. Asmentioned above, the reconstructed mapped environment may include animage converted from ultrasound of user 118's head, user 118'sshoulders, user 118's chest, and the remaining body sections down touser 118's feet. Each of these ultrasound images reconstructed in themapped environment may include a severity indicator. For instance, foruser 124 that broke his arm, the severity indicator corresponding to thehead of user 124 may be zero, the severity indicator corresponding tothe shoulder of user 124 may be one, the severity indicatorcorresponding to the arms of user 124 may be ten, and the severityindicator corresponding to the legs of user 124 may be two. Thisreconstructed mapped environment is provided to the first respondersystem 140 to facilitate determining an injury of the user, such as user124. In some implementations, the first responder system 140 may bepolice officers, firefighters, paramedics, and emergency medicaltechnicians, to name a few examples.

FIG. 2 is a contextual diagram of an example system of a buildingdestruction environment 200 for detecting one or more injuredindividuals. The building destruction environment 200 includes ademolished building 202 as a result of a natural disaster, such as anearthquake. The demolished building 202 includes one or more trappedindividuals that may have life threatening injuries. For instance, thedemolished building 202 includes user 204 lying down on the second floorof the demolished building 202 and user 206 lying under the rubble atthe bottom of the demolished building 202. In some implementations, afirst responder, such as a firefighter or a police officer, may letdrone 208 fly around a path 210 around the demolished building 202 tofind the one or more trapped individuals to detect their health status.

FIG. 2 is similar to FIG. 1 without the inclusion of a control unitserver 104 and one or more sensors at the demolished building 202. Theonly data provided to the remote processing unit 226 includes dataretrieved from the drone 208 itself. In addition, the drone 208 can scanalong path 210 until retrieved by a first responder via a client device.

During stage (A′), which is similar to stage (A) of FIG. 1, the drone208 flies a path 210 to find one or more individuals trapped in thedemolished building 202. In some implementations, the path 210 may bepreprogrammed by the first responder located at the scene of thebuilding destruction environment 200. In other implementations, the path210 may be a random path taken by the drone 208 around the demolishedbuilding 202. The drone 208 may fly the path 210 until a first responderretrieves the drone 208. In some implementations, the drone 208 may flythe path 210 until the first responder or first responder system 230receives an indication from the remote processing unit 226 indicating alocation of the one or more individuals in the demolished building 202and a corresponding health status of the located one or moreindividuals.

During stages (B′) and (C′), which are similar to stages (B) and (C) ofFIG. 1, the drone 208 detects user 204 and user 206 in the demolishedbuilding 202, as illustrated by the arrows of detected person 212.Initially, the drone 208 utilizes the camera and GPS device from the setof sensors onboard the drone 208 to detect user 204 and 206 in thedemolished building 202. The drone 208 utilizes a local processingengine that uses one or more machine learning algorithms to detectindividuals from the captured images. Once the local processing engineidentifies one or more individuals in the captured images, the localprocessing engine tags the individuals in the image with GPS locationaldata from the GPS device. The GPS locational data describes thelocational position of the detected individual. For instance, the drone208 calculates the locational position of the detected individual usingthe GPS locational position of the drone 208, the altitude of the drone208, and an estimated distance between the drone 208 and the detectedindividual using slope estimation.

During stage (D′), the drone 208 moves closer to a detected individualto perform an ultrasound scan. In order to ensure a high qualityultrasound results, the drone 208 may be programmed to move as close aspossible to the detected individual, such as user 206 collapsed underthe rubble. The drone 208 may perform a full body ultrasound scan tocapture all features of user 206. In some implementations, one or moreportions of user 206's body may be covered by rubble. The drone 208 mayonly perform scans on the exposed portion of user 206's body. Followingthe ultrasound scans of the user 206's body, the drone 208 may move tothe next detected individual, such as user 204, to perform theultrasound scan on user 204. In some implementations, the drone 208 mayreceive an audible sound coming from the user 204 while performing theultrasound scan. If the drone 208 determines the audible sound isgreater than a threshold level, such as the user 204 is screaming ormoaning in pain, the drone 208 can include an emergency request of theuser 204 in danger in the data to provide to the remote processing unit226. In addition, the drone 208 can initiate communication with a firstresponder system 230 if the drone 208 determines the user 204 is insevere danger based on the audible sound being greater than thethreshold level. Alternatively, the drone 208 can provide an indicationto the user 204 to keep calm. For instance, the drone 208 can play acalming song or the drone 208 can play an audible message to the user204 that recites “Please remain calm, help is on the way.” The drone 208may recite other messages to the user 204. Alternatively, the drone 208may cease performing ultrasound scan if the drone 208 determines theuser 204 is scared. Afterwards, the drone 208 may return to the path 210to find any other individuals in the demolished building 202.

During stage (E′), the drone 208 transmits data to the remote processingunit 226. The data includes detected person data 216, ultrasound data218, location data 220, and detected image data 222. The detected persondata 216 includes information corresponding to the number of individualsdetected during the drone 208's scan on path 210. For example, thedetected person data 216 may indicate that two individuals, user 204 anduser 206, were detected in the demolished building 202. The ultrasounddata 218 may include the ultrasound scans of the exposed body portionsof user 204 and user 206. The location data 220 may include the GPSlocational data of user 204 and user 206. The detected image data 222may include the images from the drone 208's camera that include thedetected individuals and non-detected images. In some implementations,the images may include a tag indicating whether an individual isdetected or not detected in that image.

During stage (F′), which is similar to stage (E) of FIG. 1, the remoteprocessing engine in the remote processing unit 226 processes thedetected person data 216, the ultrasound data 218, the location data220, and the detected image data 222 to produce an indication of thehealth of the one or more detected individuals.

During stage (G′), which is similar to stage (G) of FIG. 1, the remoteprocessing engine provides a notification 228 to the first respondersystem 230. As mentioned earlier, the notification includes areconstructed mapped environment of the images of the ultrasound scansand a corresponding severity indicator for each of the images.

In another exemplary use case, a drone, such as drone 208, can fly aparticular path around a vehicular accident to locate one or moreindividuals trapped in the vehicles. The drone 208 may or may not beprogrammed with a predetermined path 210 by a first responder. Inparticular, the drone 208 can be programmed to monitor an area thatincludes the vehicular accident. For example, the drone 208 can flyabove the vehicular accident, near the windows of the vehicles involvedin the accident, and low to the ground to search underneath the vehiclesto determine whether an individual has been trapped underneath thevehicle. The drone 208 can perform steps similar to that of FIG. 1 andFIG. 2 to notify first responders if one or more injured individuals arefound.

In another exemplary use case, drone 208 can fly a particular patharound a search and rescue area in a forest to locate one or more lostindividuals. The drone 208 may or may not be programmed with apredetermined path 210 by a first responder to fly through the forestsearching for the lost individuals. If the drone 208 detects a lostindividual, the drone 208 can perform steps similar to that of FIG. 1and FIG. 2 to notify first responders and determine if the detectedindividual is injured.

FIG. 3 is a contextual diagram of an example system 300 for training aneural network model for ultrasound analytics. The system 300 can trainother types of machine learning models for ultrasound analytics, such asone or clustering models, one or more deep learning models, Bayesianlearning models, or any other type of model. Briefly, and as describedin more detail below, the system 300 illustrates the application of aneural network model in the local processing engine of the drone 130 andthe application of a neural network model in the remote processingengine of the remote processing unit 136. In some implementations, thedata provided as input to the model in the local processing engine comesfrom the set of sensors 131 mounted on the drone 314. In someimplementations, the data provided as input to the model in the remoteprocessing engine comes from an output of analyzing the sensor dataprocessed by the local processing engine.

In some implementations, the local processing engine in the drone 314trains a neural network model while the drone 314 is offline. The neuralnetwork model may include an input layer, an output layer, and one ormore hidden layers. The local processing engine may use a machinelearning technique to continuously train the neural network model. Thelocal processing engine trains its neural network model using one ormore training techniques. For instance, the local processing engine maytrain the neural network model using images that include zero or moreindividuals and a tag as to whether or not an individual exists in theimage. The local processing engine applies the neural network model oncesufficiently trained.

In some implementations, the local processing engine in the drone 314applies images captured from the camera mounted on the drone 130 to thetrained model 304. The drone 314 sequentially inputs each image302A-302N to the trained model 304 at a predetermined time interval. Forinstance, the predetermined time interval may be the length of time ittakes for the trained model 304 to process one image 302C. In anotherinstance, the predetermined time interval may be spaced by a time, suchas 2 seconds.

In some implementations, the trained model 304 produces an output foreach image input to the trained model 304. The output of the trainedmodel 304 includes a detection or non-detection 306 and the input image302N. The detection or non-detection 306 includes an indication ofwhether a person is detected in the image 302N. If a person is notdetected in an image, such as image 302N, the local processing enginetags the image as no individual detected. Alternatively, if the localprocessing engine indicates a detection in 306, the image 302N isprovided as input to the location detection 310. In the locationdetection 310, the local processing engine calculates the locationalposition of the detected individual using the GPS location position ofthe drone 314, the altitude of the drone 314, and an estimated distancebetween the drone 314 and the detected individual using slopeestimation. The image 302N is tagged with the locational position of thedetected individual.

In some implementations, the local processing engine instructs the drone314 to perform an ultrasound scan at the locational position of thedetected individual, such as user 316, based on the determination thatthe image 302N includes user 316. The drone 314 moves in proximity tothe location of the user 316 and performs ultrasound scans of the user316 over different portions of the user 316's body. For instance, thedrone 314 may initiate scanning user 316's head, then move to scan theuser 316's shoulders, and down to user 316's feet to capture allfeatures of user 316. This ensures all parts of user 316 can be checkedfor a health status.

After performing the ultrasound scans, the drone 314 provides thecaptured data to a remote processing unit 324. As mentioned earlier inFIG. 2, the drone 314 provides the detected person data 318, theultrasound data 320, the location data 322, and the detected image data308 to the remote processing unit 324. In some implementations, thedrone 314 provides a new set of detected person data 318, ultrasounddata 320, location data 322, and detected image data 308 each time a newultrasound scan is performed on a newly detected individual. In otherimplementations, the drone 314 provides a new set of data each time thedrone 314 comes in contact with the charging station 142. Astransmission of data to the control unit server 104 or the remoteprocessing unit 324 draws battery usage that may be used for otherpurposes, such as flying or providing power to the set of device 131mounted on-board the drone 314, the drone 314 may be configured to onlytransmit data when connected to the charging station 142 to preservebattery life when monitoring the residential facility 102.

In some implementations, the remote processing unit 324 receives thedetected person data 318, the ultrasound data 320, the location data322, and the detected image data 308. The remote processing engine inthe remote processing unit 324 processes each of the received datapieces. Initially, the remote processing engine provides the ultrasounddata 320 to a reconstruction mechanism 328. First, the reconstructionmechanism 328 converts each scan of ultrasound into an image 329. Forexample, if the drone 314 performs ten ultrasound scans on user 316,then the reconstruction mechanism 316 converts the ten ultrasound scansto ten corresponding images.

In some implementations, the remote processing engine provides eachimage 329 converted from an ultrasound scan to a trained neural networkmodel 330. The trained model 330 is similar to trained model 304. Inparticular, the trained model 330 may include an input layer, an outputlayer, and one or more hidden layers. The remote processing engine mayuse a machine learning technique to continuously train the neuralnetwork model to create the trained model 330. The remote processingengine applies the trained model 330 once sufficiently trained.

In some implementations, the remote processing engine in the remoteprocessing unit 324 applies images 329 of the ultrasound data and thedetected person data 318 to the trained model 330. The trained model 330is trained to produce an indication 331 of the health of the individualdetected in the image from the captured ultrasound. For example, thehealth of the individual 316 may include indicating whether theindividual has sustained one or more broken bones, any externalbleeding, or burn marks, to name a few examples. The remote processingengine may tag the input image 329 with the indication 331.

In some implementations, the remote processing engine may provide thetagged input image 329 with the indication 331 output from the trainedmodel 330 to a severity indicator mechanism 332. The severity indicatormechanism 332 analyzes the tagged description 331 to determine aseverity indicator 333 of the individual in the image 329. For instance,the severity indicator 333 indicates a number that indicates theseverity of the individual's health according to the tagged description.For instance, if the tagged description indicated “external bleeding,”the severity indicator mechanism 332 may provide a severity indicationof ten. In another instance, if the tagged description indicated “brokenarm,” the severity indicator mechanism 332 may provide a severityindication of seven. This is because an external bleeding symptom may bemore severe than a broken arm, depending on the severity of the externalbleeding.

In some implementations, the severity indicator mechanism 332reconstructs a mapped environment 334 using the images converted fromthe ultrasound scans and the corresponding severity indicator for eachof the images. For example, the severity indicator mechanism 332reconstructs the mapped environment of the images of the ultrasound scanperformed on user 316. The reconstructed mapped environment 334 mayinclude an image converted from ultrasound of user 316's head, user316's shoulders, user 316's chest, and the remaining body sections downto user 316's feet. Each of these images reconstructed in the mappedenvironment may include a severity indicator 333. For instance, for user316 who may have a broken leg, the severity indicator mechanism 332 maydesignate a severity indicator of zero to the head of user 316, aseverity indicator of one corresponding to the shoulder of user 316, aseverity indicator of zero corresponding to the arms of user 316, and aseverity indicator of ten corresponding to the legs of user 316. Theremote processing engine provides the reconstructed map 334 to the firstresponder system 335 to facilitate in determining an injury of anidentified user.

In some implementations, the first responder system 335 can furthertrain the trained model 330. For instance, after the first respondersystem 335 receives the reconstructed map 334, an individual, such as amedic, of the first responder system 335 may determine that the user 316does not in fact have a broken leg, as determined by the trained model330. In response, the medic of the first responder system 335 can updateone or more medical reports that the trained model 330 accesses togenerate a reconstructed mapped environment 334 to reflect a change tothe medical diagnosis of the leg of user 316.

In some implementations, the first responder system 335 may store themedical reports and transfer the medical records to the remoteprocessing unit 226. The remote processing engine may access the medicalrecords for retraining the trained model 330. For instance, rather thanthe medical diagnosis indicating the leg of user 316 as being broken,the medical diagnosis in the medial reports indicates that the user316's leg is healthy. The trained model 330 can access the receivedupdated reports and the corresponding image 329 used in thereconstructed mapped environment 334 to retrain the trained model 330 toidentify that the leg of user 316 in the image 329 is not broken. Thetrained model 330 can be retrained with other medical diagnosis updatesfor user 316 and other users.

FIG. 4 is a flowchart of an example process 400 for providing datacorresponding to a detected individual for ultrasound analytics.Generally, the process 400 includes determining an indication of anindividual in a frame of image data; determining a location of theidentified individual in the frame of data using locational coordinates;obtaining ultrasound data of the identified individual in response to adrone's movement in proximity to the location of the identifiedindividual to capture the ultrasound data; and, providing theidentification of the individual, the location of the identifiedindividual, the frame of image data, and the ultrasound data of theidentified individual to a remote processing unit.

During 402, the drone 130 determines an identification of an individualin a frame of image data. The drone 130's set of devices 131 capturesdata during the drone 130's flight around the predetermined path 132.The data includes camera images and GPS locational data. The drone 130feeds the camera images and the GPS locational data to a localprocessing engine included in the drone 130's memory. The localprocessing engine produces an indication that an individual has beendetected in the camera images. In particular, the local processingengine in the drone 340 applies images captured from the camera mountedon the drone 130 to a trained neural network model 304. The trainedneural network model 304 produces an output for each image thatindicates a detection of a person or a non-detection of a person in theimage.

During 404, the local processing engine determines a location of theidentified individual in the frame of data using locational coordinates.In some implementations, the local processing engine calculates thelocational position of the detected individual using the GPS locationposition of the drone 314, the altitude of the drone 314, and anestimated distance between the drone 314 and the detected individualusing slope estimation. The image 302N is tagged with the locationalposition of the detected individual.

During 406, the local processing engine obtains ultrasound data of theidentified individual in response to drone 130's movement in proximityto the location of the identified individual to capture the ultrasounddata. In some implementations, the local processing engine instructs thedrone 314 to perform an ultrasound scan at the locational position ofthe detected individual, such as user 316, based on the determinationthe image 302N detects the user 316. The drone 314 moves in proximity tothe position of the user 316 and performs ultrasound scans of the user316 over different portions of the user 316's body. For instance, thedrone 314 may initiate scanning user 316's head, then move to scan theuser 316's shoulders, and proceed down to user 316's feet to capture allfeatures of user 316. This ensures all parts of user 316 can be checkedfor a health status.

During 408, the local processing engine provides the identification ofthe individual, the location of the identified individual, the frame ofimage data, and the ultrasound data of the identified individual to aremote processing unit. In some implementations, the drone 314 transmitsthe detected person data 318, the ultrasound data 320, the location data322, and the detected image data 308 to the remote processing unit 324.In some implementations, the drone 314 provides a new set of detectedperson data 318, ultrasound data 320, location data 322, and detectedimage data 308 each time a new ultrasound scan is performed on a newlydetected individual. The detected person data 318 includes informationcorresponding to the number of individuals detected during the drone314's scan on path. The location data 322 may include the GPS locationaldata of user 316. The detected image data 308 may include the imagesfrom the drone 314's camera that include the detected individuals andnon-detected images. In some implementations, the images may include atag indicating whether an individual is detected or not detected in thatimage.

FIG. 5 is a flowchart of an example 500 for processing datacorresponding to a detected individual for ultrasound analytics.Generally, the process 500 includes obtaining an identification of anindividual, a location of the identified individual, a frame of imagedata, and ultrasound data of the identified individual from a drone;generate an ultrasound image from obtained ultrasound data; determinewhether the ultrasound image includes the identified individual ashaving an injury; generate a severity indicator corresponding to each ofthe ultrasound images; generate a mapped environment that includes theultrasound images stitched together that includes the correspondingseverity indicator for each of the ultrasound images; and, providing themapped environment to a first responder system.

During 502, the remote processing engine obtains an identification of anindividual, a location of the identified individual, a frame of imagedata, and ultrasound data of the identified individual from a drone 130.In some implementations, the remote processing unit 324 receives thedetected person data 318, the ultrasound data 320, the location data322, and the detected image data 308. The remote processing engine inthe remote processing unit 324 processes each of the received dataitems.

During 504, the remote processing engine generates an ultrasound imagefrom the obtained ultrasound data. In some implementations, the remoteprocessing engine provides the ultrasound data 320 to a reconstructionmechanism 328. First, the reconstruction mechanism 328 may convert eachscan of ultrasound into an image 329. For example, if the drone 314performs ten ultrasound scans on user 316, then the reconstructionmechanism 316 converts the ten ultrasound scans to ten correspondingimages.

During 506, the remote processing engine determines whether theultrasound image includes the identified individual as having an injury.In some implementations, the remote processing engine provides eachimage converted from an ultrasound scan to a trained neural networkmodel 330. The trained model 330 is trained to produce an indication 331of the health of the individual detected in the image from the capturedultrasound. For example, the health of the individual 316 may include anindication of whether the individual has sustained one or more brokenbones, any external bleeding, or burn marks, to name a few examples. Theremote processing engine may tag the input image 329 with the indication331.

During 508, the remote processing engine generates a severity indicatorcorresponding to each of the ultrasound images. In some implementations,the remote processing engine may provide the tagged input image 329 withthe indication 331 output from the trained model 330 to a severityindicator mechanism 332. The severity indicator mechanism 332 analyzesthe tagged description 331 to determine a severity indicator 333 of theindividual in the image 329. For instance, the severity indicator 333indicates a number that indicates the severity of the individual'shealth according to the tagged description. For instance, if the taggeddescription indicated “external bleeding,” the severity indicatormechanism 332 may provide a severity indication of ten. In anotherinstance, if the tagged description indicated “broken arm,” the severityindicator mechanism 332 may provide a severity indication of seven. Thisis because an external bleeding symptom may be more severe than a brokenarm, depending on the severity of the external bleeding.

During 510, the remote processing engine generates a mapped environmentthat includes the ultrasound images stitched together that includes thecorresponding severity indicator for each of the ultrasound images. Insome implementations, the severity indicator mechanism 332 reconstructsa mapped environment 334 using the images converted from the ultrasoundscans and the corresponding severity indicator for each of the images.For example, the severity indicator mechanism 332 reconstructs themapped environment of the images of the ultrasound scan performed onuser 316. The reconstructed mapped environment 334 may include an imageconverted from ultrasound of user 316's head, user 316's shoulders, user316's chest, and the remaining body sections down to user 316's feet.Each of these images reconstructed in the mapped environment may includea severity indicator 333. For instance, for user 316 who may have abroken leg, the severity indicator mechanism 332 may designate aseverity indicator of zero to the head of user 316, a severity indicatorof one corresponding to the shoulder of user 316, a severity indicatorof zero corresponding to the arms of user 316, and a severity indicatorof ten corresponding to the legs of user 316.

During 512, the remote processing engine provides the mapped environmentto a first responder system. In some implementations, providing thereconstructed mapped environment 334 to the first responder system 335facilitates in determining an injury of an identified user.

FIG. 6 is a block diagram of an example integrated security environment600 for ultrasound analytics that may utilize various components. Theelectronic system 600 includes a network 605, a control unit 610, one ormore user devices 640 and 650, a monitoring application server 660, anda central alarm station server 670. In some examples, the network 605facilitates communications between the control unit 610, the one or moreuser devices 640 and 650, the monitoring application server 660, and thecentral alarm station server 670.

The network 605 is configured to enable exchange of electroniccommunications between devices connected to the network 605. Forexample, the network 605 may be configured to enable exchange ofelectronic communications between the control unit 610, the one or moreuser devices 640 and 650, the monitoring application server 660, and thecentral alarm station server 670. The network 605 may include, forexample, one or more of the Internet, Wide Area Networks (WANs), LocalArea Networks (LANs), analog or digital wired and wireless telephonenetworks (e.g., a public switched telephone network (PSTN), IntegratedServices Digital Network (ISDN), a cellular network, and DigitalSubscriber Line (DSL)), radio, television, cable, satellite, or anyother delivery or tunneling mechanism for carrying data. Network 605 mayinclude multiple networks or subnetworks, each of which may include, forexample, a wired or wireless data pathway. The network 605 may include acircuit-switched network, a packet-switched data network, or any othernetwork able to carry electronic communications (e.g., data or voicecommunications). For example, the network 605 may include networks basedon the Internet protocol (IP), asynchronous transfer mode (ATM), thePSTN, packet-switched networks based on IP, X.25, or Frame Relay, orother comparable technologies and may support voice using, for example,VoIP, or other comparable protocols used for voice communications. Thenetwork 605 may include one or more networks that include wireless datachannels and wireless voice channels. The network 605 may be a wirelessnetwork, a broadband network, or a combination of networks including awireless network and a broadband network.

The control unit 610 includes a controller 612 and a network module 614.The controller 612 is configured to control a control unit monitoringsystem (e.g., a control unit system) that includes the control unit 610.In some examples, the controller 612 may include a processor or othercontrol circuitry configured to execute instructions of a program thatcontrols operation of a control unit system. In these examples, thecontroller 612 may be configured to receive input from sensors, flowmeters, or other devices included in the control unit system and controloperations of devices included in the household (e.g., speakers, lights,doors, etc.). For example, the controller 612 may be configured tocontrol operation of the network module 614 included in the control unit610.

The network module 614 is a communication device configured to exchangecommunications over the network 605. The network module 614 may be awireless communication module configured to exchange wirelesscommunications over the network 605. For example, the network module 614may be a wireless communication device configured to exchangecommunications over a wireless data channel and a wireless voicechannel. In this example, the network module 614 may transmit alarm dataover a wireless data channel and establish a two-way voice communicationsession over a wireless voice channel. The wireless communication devicemay include one or more of a LTE module, a GSM module, a radio modem,cellular transmission module, or any type of module configured toexchange communications in one of the following formats: LTE, GSM orGPRS, CDMA, EDGE or EGPRS, EV-DO or EVDO, UMTS, or IP.

The network module 614 also may be a wired communication moduleconfigured to exchange communications over the network 605 using a wiredconnection. For instance, the network module 614 may be a modem, anetwork interface card, or another type of network interface device. Thenetwork module 614 may be an Ethernet network card configured to enablethe control unit 610 to communicate over a local area network and/or theInternet. The network module 614 also may be a voiceband modemconfigured to enable the alarm panel to communicate over the telephonelines of Plain Old Telephone Systems (POTS).

The control unit system that includes the control unit 610 includes oneor more sensors. For example, the monitoring system may include multiplesensors 620. The sensors 620 may include a lock sensor, a contactsensor, a motion sensor, or any other type of sensor included in acontrol unit system. The sensors 620 also may include an environmentalsensor, such as a temperature sensor, a water sensor, a rain sensor, awind sensor, a light sensor, a smoke detector, a carbon monoxidedetector, an air quality sensor, etc. The sensors 620 further mayinclude a health monitoring sensor, such as a prescription bottle sensorthat monitors taking of prescriptions, a blood pressure sensor, a bloodsugar sensor, a bed mat configured to sense presence of liquid (e.g.,bodily fluids) on the bed mat, etc. In some examples, the sensors 620may include a radio-frequency identification (RFID) sensor thatidentifies a particular article that includes a pre-assigned RFID tag.

The control unit 610 communicates with the module 622 and the camera 630to perform monitoring. The module 622 is connected to one or moredevices that enable home automation control. For instance, the module622 may be connected to one or more lighting systems and may beconfigured to control operation of the one or more lighting systems.Also, the module 622 may be connected to one or more electronic locks atthe property and may be configured to control operation of the one ormore electronic locks (e.g., control Z-Wave locks using wirelesscommunications in the Z-Wave protocol. Further, the module 622 may beconnected to one or more appliances at the property and may beconfigured to control operation of the one or more appliances. Themodule 622 may include multiple modules that are each specific to thetype of device being controlled in an automated manner. The module 622may control the one or more devices based on commands received from thecontrol unit 610. For instance, the module 622 may cause a lightingsystem to illuminate an area to provide a better image of the area whencaptured by a camera 630.

The camera 630 may be a video/photographic camera or other type ofoptical sensing device configured to capture images. For instance, thecamera 630 may be configured to capture images of an area within abuilding or within a residential facility 102 monitored by the controlunit 610. The camera 630 may be configured to capture single, staticimages of the area and also video images of the area in which multipleimages of the area are captured at a relatively high frequency (e.g.,thirty images per second). The camera 630 may be controlled based oncommands received from the control unit 610.

The camera 630 may be triggered by several different types oftechniques. For instance, a Passive Infra-Red (PIR) motion sensor may bebuilt into the camera 630 and used to trigger the camera 630 to captureone or more images when motion is detected. The camera 630 also mayinclude a microwave motion sensor built into the camera and used totrigger the camera 630 to capture one or more images when motion isdetected. The camera 630 may have a “normally open” or “normally closed”digital input that can trigger capture of one or more images whenexternal sensors (e.g., the sensors 620, PIR, door/window, etc.) detectmotion or other events. In some implementations, the camera 630 receivesa command to capture an image when external devices detect motion oranother potential alarm event. The camera 630 may receive the commandfrom the controller 612 or directly from one of the sensors 620.

In some examples, the camera 630 triggers integrated or externalilluminators (e.g., Infra-Red, Z-wave controlled “white” lights, lightscontrolled by the module 622, etc.) to improve image quality when thescene is dark. An integrated or separate light sensor may be used todetermine if illumination is desired and may result in increased imagequality.

The camera 630 may be programmed with any combination of time/dayschedules, system “arming state”, or other variables to determinewhether images should be captured or not when triggers occur. The camera630 may enter a low-power mode when not capturing images. In this case,the camera 630 may wake periodically to check for inbound messages fromthe controller 612. The camera 630 may be powered by internal,replaceable batteries if located remotely from the control unit 610. Thecamera 630 may employ a small solar cell to recharge the battery whenlight is available. Alternatively, the camera 630 may be powered by thecontroller's 612 power supply if the camera 630 is co-located with thecontroller 612.

In some implementations, the camera 630 communicates directly with themonitoring application server 660 over the Internet. In theseimplementations, image data captured by the camera 630 does not passthrough the control unit 610 and the camera 630 receives commandsrelated to operation from the monitoring application server 660.

The system 600 also includes thermostat 634 to perform dynamicenvironmental control at the property. The thermostat 634 is configuredto monitor temperature and/or energy consumption of an HVAC systemassociated with the thermostat 634, and is further configured to providecontrol of environmental (e.g., temperature) settings. In someimplementations, the thermostat 634 can additionally or alternativelyreceive data relating to activity at a property and/or environmentaldata at a property, e.g., at various locations indoors and outdoors atthe property. The thermostat 634 can directly measure energy consumptionof the HVAC system associated with the thermostat, or can estimateenergy consumption of the HVAC system associated with the thermostat634, for example, based on detected usage of one or more components ofthe HVAC system associated with the thermostat 634. The thermostat 634can communicate temperature and/or energy monitoring information to orfrom the control unit 610 and can control the environmental (e.g.,temperature) settings based on commands received from the control unit610.

In some implementations, the thermostat 634 is a dynamicallyprogrammable thermostat and can be integrated with the control unit 610.For example, the dynamically programmable thermostat 634 can include thecontrol unit 610, e.g., as an internal component to the dynamicallyprogrammable thermostat 634. In addition, the control unit 610 can be agateway device that communicates with the dynamically programmablethermostat 634.

A module 637 is connected to one or more components of an HVAC systemassociated with a property, and is configured to control operation ofthe one or more components of the HVAC system. In some implementations,the module 637 is also configured to monitor energy consumption of theHVAC system components, for example, by directly measuring the energyconsumption of the HVAC system components or by estimating the energyusage of the one or more HVAC system components based on detecting usageof components of the HVAC system. The module 637 can communicate energymonitoring information and the state of the HVAC system components tothe thermostat 634 and can control the one or more components of theHVAC system based on commands received from the thermostat 634.

In some examples, the system 600 further includes one or more roboticdevices. The robotic devices may be any type of robots that are capableof moving and taking actions that assist in security monitoring. Forexample, the robotic devices may include drones that are capable ofmoving throughout a property based on automated control technologyand/or user input control provided by a user. In this example, thedrones may be able to fly, roll, walk, or otherwise move about theproperty. The drones may include helicopter type devices (e.g., quadcopters), rolling helicopter type devices (e.g., roller copter devicesthat can fly and also roll along the ground, walls, or ceiling) and landvehicle type devices (e.g., automated cars that drive around aproperty). In some cases, the robotic devices may be robotic devicesthat are intended for other purposes and merely associated with thesystem 600 for use in appropriate circumstances. For instance, a roboticvacuum cleaner device may be associated with the monitoring system 600as one of the robotic devices and may be controlled to take actionresponsive to monitoring system events.

In some examples, the robotic devices automatically navigate within aproperty. In these examples, the robotic devices include sensors andcontrol processors that guide movement of the robotic devices within theproperty. For instance, the robotic devices may navigate within theproperty using one or more cameras, one or more proximity sensors, oneor more gyroscopes, one or more accelerometers, one or moremagnetometers, a global positioning system (GPS) unit, an altimeter, oneor more sonar or laser sensors, and/or any other types of sensors thataid in navigation about a space. The robotic devices may include controlprocessors that process output from the various sensors and control therobotic devices to move along a path that reaches the desireddestination and avoids obstacles. In this regard, the control processorsdetect walls or other obstacles in the property and guide movement ofthe robotic devices in a manner that avoids the walls and otherobstacles.

In addition, the robotic devices may store data that describesattributes of the property. For instance, the robotic devices may storea floorplan and/or a three-dimensional model of the property thatenables the robotic devices to navigate the property. During initialconfiguration, the robotic devices may receive the data describingattributes of the property, determine a frame of reference to the data(e.g., a home or reference location in the property), and navigate theproperty based on the frame of reference and the data describingattributes of the property. Further, initial configuration of therobotic devices also may include learning of one or more navigationpatterns in which a user provides input to control the robotic devicesto perform a specific navigation action (e.g., fly to an upstairsbedroom and spin around while capturing video and then return to a homecharging base). In this regard, the robotic devices may learn and storethe navigation patterns such that the robotic devices may automaticallyrepeat the specific navigation actions upon a later request.

In some examples, the robotic devices may include data capture andrecording devices. In these examples, the robotic devices may includeone or more cameras, one or more motion sensors, one or moremicrophones, one or more biometric data collection tools, one or moretemperature sensors, one or more humidity sensors, one or more air flowsensors, and/or any other types of sensors that may be useful incapturing monitoring data related to the property and users in theproperty. The one or more biometric data collection tools may beconfigured to collect biometric samples of a person in the home with orwithout contact of the person. For instance, the biometric datacollection tools may include a fingerprint scanner, a hair samplecollection tool, a skin cell collection tool, and/or any other tool thatallows the robotic devices to take and store a biometric sample that canbe used to identify the person (e.g., a biometric sample with DNA thatcan be used for DNA testing).

In some implementations, the robotic devices may include output devices.In these implementations, the robotic devices may include one or moredisplays, one or more speakers, and/or any type of output devices thatallow the robotic devices to communicate information to a nearby user.

The robotic devices also may include a communication module that enablesthe robotic devices to communicate with the control unit 610, eachother, and/or other devices. The communication module may be a wirelesscommunication module that allows the robotic devices to communicatewirelessly. For instance, the communication module may be a Wi-Fi modulethat enables the robotic devices to communicate over a local wirelessnetwork at the property. The communication module further may be a 900MHz wireless communication module that enables the robotic devices tocommunicate directly with the control unit 610. Other types ofshort-range wireless communication protocols, such as Bluetooth,Bluetooth LE, Zwave, Zigbee, etc., may be used to allow the roboticdevices to communicate with other devices in the property.

The robotic devices further may include processor and storagecapabilities. The robotic devices may include any suitable processingdevices that enable the robotic devices to operate applications andperform the actions described throughout this disclosure. In addition,the robotic devices may include solid state electronic storage thatenables the robotic devices to store applications, configuration data,collected sensor data, and/or any other type of information available tothe robotic devices.

The robotic devices are associated with one or more charging stations.The charging stations may be located at predefined home base orreference locations in the property. The robotic devices may beconfigured to navigate to the charging stations after completion oftasks needed to be performed for the monitoring system 600. Forinstance, after completion of a monitoring operation or upon instructionby the control unit 610, the robotic devices may be configured toautomatically fly to and land on one of the charging stations. In thisregard, the robotic devices may automatically maintain a fully chargedbattery in a state in which the robotic devices are ready for use by themonitoring system 600.

The charging stations may be contact based charging stations and/orwireless charging stations. For contact based charging stations, therobotic devices may have readily accessible points of contact that therobotic devices are capable of positioning and mating with acorresponding contact on the charging station. For instance, ahelicopter type robotic device may have an electronic contact on aportion of its landing gear that rests on and mates with an electronicpad of a charging station when the helicopter type robotic device landson the charging station. The electronic contact on the robotic devicemay include a cover that opens to expose the electronic contact when therobotic device is charging and closes to cover and insulate theelectronic contact when the robotic device is in operation.

For wireless charging stations, the robotic devices may charge through awireless exchange of power. In these cases, the robotic devices needonly locate themselves closely enough to the wireless charging stationsfor the wireless exchange of power to occur. In this regard, thepositioning needed to land at a predefined home base or referencelocation in the property may be less precise than with a contact basedcharging station. Based on the robotic devices landing at a wirelesscharging station, the wireless charging station outputs a wirelesssignal that the robotic devices receive and convert to a power signalthat charges a battery maintained on the robotic devices.

In some implementations, each of the robotic devices has a correspondingand assigned charging station such that the number of robotic devicesequals the number of charging stations. In these implementations, therobotic devices always navigate to the specific charging stationassigned to that robotic device. For instance, a first robotic devicemay always use a first charging station and a second robotic device mayalways use a second charging station.

In some examples, the robotic devices may share charging stations. Forinstance, the robotic devices may use one or more community chargingstations that are capable of charging multiple robotic devices. Thecommunity charging station may be configured to charge multiple roboticdevices in parallel. The community charging station may be configured tocharge multiple robotic devices in serial such that the multiple roboticdevices take turns charging and, when fully charged, return to apredefined home base or reference location in the property that is notassociated with a charger. The number of community charging stations maybe less than the number of robotic devices.

Also, the charging stations may not be assigned to specific roboticdevices and may be capable of charging any of the robotic devices. Inthis regard, the robotic devices may use any suitable, unoccupiedcharging station when not in use. For instance, when one of the roboticdevices has completed an operation or is in need of battery charge, thecontrol unit 610 references a stored table of the occupancy status ofeach charging station and instructs the robotic device to navigate tothe nearest charging station that is unoccupied.

The system 600 further includes one or more integrated security devices680. The one or more integrated security devices may include any type ofdevice used to provide alerts based on received sensor data. Forinstance, the one or more control units 610 may provide one or morealerts to the one or more integrated security input/output devices.Additionally, the one or more control units 610 may receive one or moresensor data from the sensors 620 and determine whether to provide analert to the one or more integrated security input/output devices 680.

The sensors 620, the module 622, the camera 630, the thermostat 634, andthe integrated security devices 680 communicate with the controller 612over communication links 624, 626, 628, 632, 684, and 686. Thecommunication links 624, 626, 628, 632, 684, and 686 may be a wired orwireless data pathway configured to transmit signals from the sensors620, the module 622, the camera 630, the thermostat 634, and theintegrated security devices 680 to the controller 612. The sensors 620,the module 622, the camera 630, the thermostat 634, and the integratedsecurity devices 680 may continuously transmit sensed values to thecontroller 612, periodically transmit sensed values to the controller612, or transmit sensed values to the controller 612 in response to achange in a sensed value.

The communication links 624, 626, 628, 632, 684, and 686 may include alocal network. The sensors 620, the module 622, the camera 630, thethermostat 634, and the integrated security devices 680, and thecontroller 612 may exchange data and commands over the local network.The local network may include 802.11 “Wi-Fi” wireless Ethernet (e.g.,using low-power Wi-Fi chipsets), Z-Wave, Zigbee, Bluetooth, “Homeplug”or other “Powerline” networks that operate over AC wiring, and aCategory 5 (CAT5) or Category 6 (CAT6) wired Ethernet network. The localnetwork may be a mesh network constructed based on the devices connectedto the mesh network.

The monitoring application server 660 is an electronic device configuredto provide monitoring services by exchanging electronic communicationswith the control unit 610, the one or more user devices 640 and 650, andthe central alarm station server 670 over the network 605. For example,the monitoring application server 660 may be configured to monitorevents (e.g., alarm events) generated by the control unit 610. In thisexample, the monitoring application server660 may exchange electroniccommunications with the network module 614 included in the control unit610 to receive information regarding events (e.g., alerts) detected bythe control unit server 104a. The monitoring application server 660 alsomay receive information regarding events (e.g., alerts) from the one ormore user devices 640 and 650.

In some examples, the monitoring application server 660 may route alertdata received from the network module 614 or the one or more userdevices 640 and 650 to the central alarm station server 670. Forexample, the monitoring application server 660 may transmit the alertdata to the central alarm station server 670 over the network 605.

The monitoring application server 660 may store sensor and image datareceived from the monitoring system and perform analysis of sensor andimage data received from the monitoring system. Based on the analysis,the monitoring application server 660 may communicate with and controlaspects of the control unit 610 or the one or more user devices 640 and650.

The central alarm station server 670 is an electronic device configuredto provide alarm monitoring service by exchanging communications withthe control unit 610, the one or more mobile devices 640 and 650, andthe monitoring application server 660 over the network 605. For example,the central alarm station server 670 may be configured to monitoralerting events generated by the control unit 610. In this example, thecentral alarm station server 670 may exchange communications with thenetwork module 614 included in the control unit 610 to receiveinformation regarding alerting events detected by the control unit 610.The central alarm station server 670 also may receive informationregarding alerting events from the one or more mobile devices 640 and650 and/or the monitoring application server 660.

The central alarm station server 670 is connected to multiple terminals672 and 674. The terminals 672 and 674 may be used by operators toprocess alerting events. For example, the central alarm station server670 may route alerting data to the terminals 672 and 674 to enable anoperator to process the alerting data. The terminals 672 and 674 mayinclude general-purpose computers (e.g., desktop personal computers,workstations, or laptop computers) that are configured to receivealerting data from a server in the central alarm station server 670 andrender a display of information based on the alerting data. Forinstance, the controller 612 may control the network module 614 totransmit, to the central alarm station server 670, alerting dataindicating that a sensor 620 detected motion from a motion sensor viathe sensors 620. The central alarm station server 670 may receive thealerting data and route the alerting data to the terminal 672 forprocessing by an operator associated with the terminal 672. The terminal672 may render a display to the operator that includes informationassociated with the alerting event (e.g., the lock sensor data, themotion sensor data, the contact sensor data, etc.) and the operator mayhandle the alerting event based on the displayed information.

In some implementations, the terminals 672 and 674 may be mobile devicesor devices designed for a specific function. Although FIG.6 illustratestwo terminals for brevity, actual implementations may include more (and,perhaps, many more) terminals.

The one or more user devices 640 and 650 are devices that host anddisplay user interfaces. For instance, the user device 640 is a mobiledevice that hosts one or more native applications (e.g., the smart homeapplication 642). The user device 640 may be a cellular phone or anon-cellular locally networked device with a display. The user device640 may include a cell phone, a smart phone, a tablet PC, a personaldigital assistant (“PDA”), or any other portable device configured tocommunicate over a network and display information. For example,implementations may also include Blackberry-type devices (e.g., asprovided by Research in Motion), electronic organizers, iPhone-typedevices (e.g., as provided by Apple), iPod devices (e.g., as provided byApple) or other portable music players, other communication devices, andhandheld or portable electronic devices for gaming, communications,and/or data organization. The user device 640 may perform functionsunrelated to the monitoring system, such as placing personal telephonecalls, playing music, playing video, displaying pictures, browsing theInternet, maintaining an electronic calendar, etc.

The user device 640 includes a smart home application 642. The smarthome application 642 refers to a software/firmware program running onthe corresponding mobile device that enables the user interface andfeatures described throughout. The user device 640 may load or installthe smart home application 642 based on data received over a network ordata received from local media. The smart home application 642 runs onmobile devices platforms, such as iPhone, iPod touch, Blackberry, GoogleAndroid, Windows Mobile, etc. The smart home application 642 enables theuser device 640 to receive and process image and sensor data from themonitoring system.

The user device 650 may be a general-purpose computer (e.g., a desktoppersonal computer, a workstation, or a laptop computer) that isconfigured to communicate with the monitoring application server 660and/or the control unit 610 over the network 605. The user device 650may be configured to display a smart home user interface 652 that isgenerated by the user device 650 or generated by the monitoringapplication server 660. For example, the user device 650 may beconfigured to display a user interface (e.g., a web page) provided bythe monitoring application server 660 that enables a user to perceiveimages captured by the camera 630 and/or reports related to themonitoring system. Although FIG. 6 illustrates two user devices forbrevity, actual implementations may include more (and, perhaps, manymore) or fewer user devices.

In some implementations, the one or more user devices 640 and 650communicate with and receive monitoring system data from the controlunit 610 using the communication link 638. For instance, the one or moreuser devices 640 and 650 may communicate with the control unit 610 usingvarious local wireless protocols such as Wi-Fi, Bluetooth, Zwave,Zigbee, HomePlug (ethernet over powerline), or wired protocols such asEthernet and USB, to connect the one or more user devices 640 and 650 tolocal security and automation equipment. The one or more user devices640 and 650 may connect locally to the monitoring system and its sensorsand other devices. The local connection may improve the speed of statusand control communications because communicating through the network 605with a remote server (e.g., the monitoring application server 660) maybe significantly slower.

Although the one or more user devices 640 and 650 are shown ascommunicating with the control unit 610, the one or more user devices640 and 650 may communicate directly with the sensors and other devicescontrolled by the control unit 610. In some implementations, the one ormore user devices 640 and 650 replace the control unit 610 and performthe functions of the control unit 610 for local monitoring and longrange/offsite communication.

In other implementations, the one or more user devices 640 and 650receive monitoring system data captured by the control unit 610 throughthe network 605. The one or more user devices 640, 650 may receive thedata from the control unit 610 through the network 605 or the monitoringapplication server 660 may relay data received from the control unit 610to the one or more user devices 640 and 650 through the network 605. Inthis regard, the monitoring application server 660 may facilitatecommunication between the one or more user devices 640 and 650 and themonitoring system.

In some implementations, the one or more user devices 640 and 650 may beconfigured to switch whether the one or more user devices 640 and 650communicate with the control unit 610 directly (e.g., through link 638)or through the monitoring application server 660 (e.g., through network605) based on a location of the one or more user devices 640 and 650.For instance, when the one or more user devices 640 and 650 are locatedclose to the control unit 610 and in range to communicate directly withthe control unit 610, the one or more user devices 640 and 650 usedirect communication. When the one or more user devices 640 and 650 arelocated far from the control unit 610 and not in range to communicatedirectly with the control unit 610, the one or more user devices 640 and650 use communication through the monitoring application server 660.

Although the one or more user devices 640 and 650 are shown as beingconnected to the network 605, in some implementations, the one or moreuser devices 640 and 650 are not connected to the network 605. In theseimplementations, the one or more user devices 640 and 650 communicatedirectly with one or more of the monitoring system components and nonetwork (e.g., Internet) connection or reliance on remote servers isneeded.

In some implementations, the one or more user devices 640 and 650 areused in conjunction with only local sensors and/or local devices in ahouse. In these implementations, the system 600 only includes the one ormore user devices 640 and 650, the sensors 620, the module 622, thecamera 630, and the robotic devices. The one or more user devices 640and 650 receive data directly from the sensors 620, the module 622, thecamera 630, and the robotic devices and sends data directly to thesensors 620, the module 622, the camera 630, and the robotic devices.The one or more user devices 640, 650 provide the appropriateinterfaces/processing to provide visual surveillance and reporting.

In other implementations, the system 600 further includes network 605and the sensors 620, the module 622, the camera 630, the thermostat 634,and the robotic devices are configured to communicate sensor and imagedata to the one or more user devices 640 and 650 over network 605 (e.g.,the Internet, cellular network, etc.). In yet another implementation,the sensors 620, the module 622, the camera 630, the thermostat 634, andthe robotic devices (or a component, such as a bridge/router) areintelligent enough to change the communication pathway from a directlocal pathway when the one or more user devices 640 and 650 are in closephysical proximity to the sensors 620, the module 622, the camera 630,the thermostat 634, and the robotic devices to a pathway over network605 when the one or more user devices 640 and 650 are farther from thesensors 620, the module 622, the camera 630, the thermostat 634, and therobotic devices. In some examples, the system leverages GPS informationfrom the one or more user devices 640 and 650 to determine whether theone or more user devices 640 and 650 are close enough to the sensors620, the module 622, the camera 630, the thermostat 634, and the roboticdevices to use the direct local pathway or whether the one or more userdevices 640 and 650 are far enough from the sensors 620, the module 622,the camera 630, the thermostat 634, and the robotic devices that thepathway over network 605 is required. In other examples, the systemleverages status communications (e.g., pinging) between the one or moreuser devices 640 and 650 and the sensors 620, the module 622, the camera630, the thermostat 634, and the robotic devices to determine whethercommunication using the direct local pathway is possible. Ifcommunication using the direct local pathway is possible, the one ormore user devices 640 and 650 communicate with the sensors 620, themodule 622, the camera 630, the thermostat 634, and the robotic devicesusing the direct local pathway. If communication using the direct localpathway is not possible, the one or more user devices 640 and 650communicate with the sensors 620, the module 622, the camera 630, thethermostat 634, and the robotic devices using the pathway over network605.

In some implementations, the system 600 provides end users with accessto images captured by the camera 630 to aid in decision making. Thesystem 600 may transmit the images captured by the camera 630 over awireless WAN network to the user devices 640 and 650. Becausetransmission over a wireless WAN network may be relatively expensive,the system 600 uses several techniques to reduce costs while providingaccess to significant levels of useful visual information.

In some implementations, a state of the monitoring system and otherevents sensed by the monitoring system may be used to enable/disablevideo/image recording devices (e.g., the camera 630). In theseimplementations, the camera 630 may be set to capture images on aperiodic basis when the alarm system is armed in an “Away” state, butset not to capture images when the alarm system is armed in a “Stay”state or disarmed. In addition, the camera 630 may be triggered to begincapturing images when the alarm system detects an event, such as analarm event, a door-opening event for a door that leads to an areawithin a field of view of the camera 630, or motion in the area withinthe field of view of the camera 630. In other implementations, thecamera 630 may capture images continuously, but the captured images maybe stored or transmitted over a network when needed.

The described systems, methods, and techniques may be implemented indigital electronic circuitry, computer hardware, firmware, software, orin combinations of these elements. Apparatus implementing thesetechniques may include appropriate input and output devices, a computerprocessor, and a computer program product tangibly embodied in amachine-readable storage device for execution by a programmableprocessor. A process implementing these techniques may be performed by aprogrammable processor executing a program of instructions to performdesired functions by operating on input data and generating appropriateoutput. The techniques may be implemented in one or more computerprograms that are executable on a programmable system including at leastone programmable processor coupled to receive data and instructionsfrom, and to transmit data and instructions to, a data storage system,at least one input device, and at least one output device. Each computerprogram may be implemented in a high-level procedural or object-orientedprogramming language, or in assembly or machine language if desired; andin any case, the language may be a compiled or interpreted language.Suitable processors include, by way of example, both general and specialpurpose microprocessors. Generally, a processor will receiveinstructions and data from a read-only memory and/or a random accessmemory. Storage devices suitable for tangibly embodying computer programinstructions and data include all forms of non-volatile memory,including by way of example semiconductor memory devices, such asErasable Programmable Read-Only Memory (EPROM), Electrically ErasableProgrammable Read-Only Memory (EEPROM), and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and Compact Disc Read-Only Memory (CD-ROM). Anyof the foregoing may be supplemented by, or incorporated in, speciallydesigned ASICs (application-specific integrated circuits).

It will be understood that various modifications may be made. Forexample, other useful implementations could be achieved if steps of thedisclosed techniques were performed in a different order and/or ifcomponents in the disclosed systems were combined in a different mannerand/or replaced or supplemented by other components. Accordingly, otherimplementations are within the scope of the disclosure.

What is claimed is:
 1. A monitoring system that is configured to monitora property, the monitoring system comprising: one or more sensors thatare located at the property and that are configured to generate firstsensor data; a monitor control unit that is configured to: receive thefirst sensor data; based on the first sensor data, generate an alarmevent for the property; and based on generating the alarm event for theproperty, dispatch an autonomous drone; and the autonomous drone that isconfigured to: navigate the property; generate, using an onboard sensor,second sensor data; based on the second sensor data, determine alocation within the property where a person is likely located; andprovide, for output, data indicating the location within the propertywhere the person is likely located.
 2. The monitoring system of claim 1,wherein the autonomous drone is configured to: based on navigating theproperty, generate a map of the property; and provide, for output, thedata indicating the location within the property where the person islikely located by providing, for output, the map of the property withthe location where the person is likely located.
 3. The monitoringsystem of claim 1, wherein the autonomous drone is configured to: basedon the second sensor data, determine that the person is likely injured;and provide, for output, data indicating that the person is likelyinjured.
 4. The monitoring system of claim 3, wherein the autonomousdrone is configured to: based on determining that the person is likelyinjured, generate using an additional onboard sensor, third sensor data;based on the third sensor data, determine a severity of the injury tothe person; and provide, for output, the data indicating that the personis likely injured by providing, for output, the data indicating that theperson is likely injured and data indicating the severity of the injuryto the person.
 5. The monitoring system of claim 4, wherein: the onboardsensor is a camera and the second sensor data is image data, and theadditional onboard sensor is an ultrasound sensor and the third sensordata is ultrasound data.
 6. The monitoring system of claim 1, whereinthe autonomous drone is configured to: provide the second sensor data asan input to a model trained to identify locations of people; anddetermine the location within the property where a person is likelylocated based on an output of the model trained to identify locations ofpeople based on the second sensor data.
 7. The monitoring system ofclaim 6, wherein the autonomous drone is configured to: receive labeledtraining data that includes first labeled sensor data that correspondslocations with people and second labeled sensor data that corresponds tolocations without people; train, using machine learning, the firstlabeled sensor data, and the second labeled sensor data, the model toidentify locations of people based on the second sensor data.
 8. Themonitoring system of claim 1, wherein the autonomous drone is configuredto: based on the second sensor data, determine that the person is likelyalive; and provide, for output, data indicating that the person islikely alive.
 9. The monitoring system of claim 8, wherein: the secondsensor is a microphone and the second sensor data is audio data, and theautonomous drone is configured to: provide the audio data as an input toa model trained to identify human sounds; and determine that the personis likely alive based on an output of the model trained to identifyhuman sounds.
 10. The monitoring system of claim 1, wherein theautonomous drone is configured to: based on determining a locationwithin the property where a person is likely located, activate acommunication channel between a device outside the property and theautonomous drone.
 11. A computer-implemented method, comprising:generating, by one or more sensors of a monitoring system that isconfigured to monitor a property, first sensor data; based on the firstsensor data, generating, by the monitoring system, an alarm event forthe property; based on generating the alarm event for the property,dispatching, by the monitoring system, an autonomous drone; navigating,by the autonomous drone of the monitoring system, the property;generating, by the autonomous drone of the monitoring system, secondsensor data; based on the second sensor data, determining, by themonitoring system, a location within the property where a person islikely located; and provide, for output by the monitoring system, dataindicating the location within the property where the person is likelylocated.
 12. The computer-implemented method of claim 11, comprising:based on navigating the property, generating, by the monitoring system,a map of the property; and providing, by the monitoring system, foroutput, data indicating the location within the property where theperson is likely located by providing, for output, the map of theproperty with the location where the person is likely located.
 13. Thecomputer-implemented method of claim 11, comprising: determining, by themonitoring system, that the person is likely injured based on secondsensor data; and providing, for output by the monitoring system, dataindicating that the person is likely injured.
 14. Thecomputer-implemented method of claim 13, comprising: based ondetermining that the person is likely injured, generating, by theautonomous drone of the monitoring system, using an additional onboardsensor, third sensor data; based on the third sensor data, determining,by the monitoring system, a severity of the injury to the person; andproviding, for output by the monitoring system, the data indicating thatthe person is likely injured by providing, for output, the dataindicating that the person is likely injured and data indicating theseverity of the injury to the person.
 15. The computer-implementedmethod of claim 14, wherein: the onboard sensor is a camera and thesecond sensor data is image data, and the additional onboard sensor isan ultrasound sensor and the third sensor data is ultrasound data. 16.The computer-implemented method of claim 11, comprising: providing, bythe autonomous drone of the monitoring system, second sensor data as aninput to a model trained to identify locations of people; anddetermining, by the monitoring system, the location within the propertywhere a person is likely located based on an output of the model trainedto identify locations of people based on the second sensor data.
 17. Thecomputer-implemented method of claim 16, comprising: receiving, by themonitoring system, labeled training data that includes first labeledsensor data that corresponds locations with people and second labeledsensor data that corresponds to locations without people; and training,by the monitoring system, using machine learning, the first labeledsensor data, and the second labeled sensor data, the model to identifylocations of people based on the second sensor data.
 18. Thecomputer-implemented method of claim 11, comprising: based on the secondsensor data, determining, by the monitoring system, that the person islikely alive; and providing, for output by the monitoring system, dataindicating that the person is likely alive.
 19. The computer-implementedmethod of claim 18, wherein: the second sensor is a microphone and thesecond sensor data is audio data, and the method comprises: providing,by the monitoring system, the audio data as an input to a model trainedto identify human sounds; and determining, by the monitoring system,that the person is likely alive based on an output of the model trainedto identify human sounds.
 20. The computer-implemented method of claim11, comprising: based on determining a location within the propertywhere a person is likely located, activating, by the monitoring system,a communication channel between a device outside the property and theautonomous drone.